The invention concerns a method for image processing, especially for converting a two-dimensional image of a three-dimensional real subject into a three-dimensional representation of the same three-dimensional real subject, wherein the subject is composed of elements in the two-dimensional image.
The object of the present invention is to provide a new and simple technique which enables colour pictures to be converted from film or video, which is a two-dimensional representation in the image plane with width and height co-ordinates (x, y), into a three-dimensional representation in space with width, height and depth co-ordinates (X, Y, Z). Such a three-dimensional representation may, for example, be used for analysing form elements in the image surface (object diagnostics) or for the production of three-dimensional images/film for a medium suited thereto.
Colour pictures from film or video are a photographic representation of light from one or more light sources with a certain spectral composition which at a given point of time struck objects in the camera""s image representation. All objects absorb certain colours and reflect the rest. The object colour is determined by the spectral distribution of the light reflected by the object. The recording on film of an object is dependent on several elements, including:
1) The spectral distribution of the light source(s).
2) The spectral reflecting power of the object.
3) The distance from the camera and absorption spectrum of this air perspective.
4) The photographic material""s colour recording ability.
In order to obtain space co-ordinates, including the depth parameter Z, for subject elements in an existing image, the image""s colour is analysed. The colour is a recording of incident light sources and reflecting objects.
The colour of light sources is measured in colour temperature and is indicated in Kelvin (K). When photographing a scenario the colour temperature of the incident and reflected light from objects in the image surface will normally be measured with a colour light meter, using film with a colour recording ability which is adapted for the purpose (daylight, artificial light, etc.). The colour temperature of light sources varies with the distance to the camera.
The individual image elements in a colour picture represent the reflecting objects in the image representation. The colour of the image elements is unambiguously determined by three measurable parameters: hue, saturation and brightness. These colour parameters vary with the distance to the camera.
The hue refers to the colour""s place in the electromagnetic spectrum, cf. Newton""s colour spectrum and circle. This is normally indicated by the relative composition of the spectral colours Red, Green and Blue (R, G, B), or alternatively Magenta, Yellow and Cyan (M, Y, C). Cf. also transformation to colour co-ordinates according to the CIE 1931 standard, where all visible colours (i.e. in the wavelength range from 400 nm to 770 nm) for a given degree of brightness can be assigned exact co-ordinates within a diagram which is defined by the spectral colours and a purple line.
The saturation indicates how much a colour has been mixed with other colours, and is designated as a percentage (%). A pure spectral colour is called saturated (degree of saturation 100%), white and grey are called unsaturated colours (degree of saturation 0%).
The brightness (light intensity) is a measure of the incident amount of light relative to reflection from an absolutely white surface (100% reflecting) and is designated thereby as a relative percentage (%).
The recorded colour of light reflected from objects which are moving at different distances relative to a film camera varies. The colour temperature of light sources varies with the distance. The individual element""s hue (RGB) is constant, but brightness and saturation vary with the distance. A high degree of brightness, saturated colours and low colour temperature occur close to the image surface (short distance), giving so-called active levels. Correspondingly, a low degree of brightness, unsaturated colours and high colour temperature will occur far from the image surface (long distance), giving passive levels. In addition this works internally in all these levels (active/passive), where warm colours appear more active in complementary contrast to cold colours. Warm colours are M, MY and Y. Cold colours are YC and C.
Due to the variation in the said characteristics with distance from a film camera, it may be deduced that each colour has its own perspective place in one and the same scenario in relation to the image""s other colours. This is the basis of the present invention.
According to the invention a method is proposed for image processing, especially for converting a two-dimensional image of a three-dimensional real subject into a three-dimensional representation of the same three-dimensional real subject, wherein the subject is composed of elements in the two-dimensional image, and wherein the two-dimensional image is provided by means of a camera. The method is characterized by:
defining a reference plane corresponding to the camera""s focal plane lying closest to the camera""s focal plane, the reference plane comprising elements corresponding to elements in the two-dimensional image,
defining colour parameters hue, saturation, brightness for each element in the reference plane,
creating a reference scale by determining values of the colour parameters by means of an image sequence of individual images, each of which depicts the subject in different predetermined focal planes, the changes in colour parameters between the respective focal planes being used for calibration of a geometric measuring scale which refers respective measured colour parameters to metric or perspective distance values,
measuring and registering colour parameters hue, saturation and brightness for each element in the two-dimensional image,
comparing colour parameters for each element in the two-dimensional picture with colour parameters for the corresponding element in the reference plane,
based on said comparison, assigning to each element in the two-dimensional image a distance value dh;ds;db, where dh results from comparing hue values, ds results from comparing saturation values and db results from comparing brightness values,
calculating the distance d(z) between the elements in the two-dimensional image and the reference plane, d(z) being the distance measured along a z-axis perpendicular to the reference plane and with origin therein, as a weighed average of the distance values dh,ds,db; d(z)=⅓*(kh*dh+ks*ds+kb*db), where kh, ks and kb constitute weight factors based on empirical data,
performing the transformations d(x)xe2x86x92X, d(y)xe2x86x92Y and d(z)xe2x86x92Z, where X,Y,Z are the element""s co-ordinates in a three-dimensional co-ordinate system with origin in the reference plane, the transformations d(x)xe2x86x92X, d(y)xe2x86x92Y being obtained by means of a known procedure for converting image elements in the plane into elements in space by means of calculated ratios (scale) between the camera""s image plane and the focal planes in the space, the closest plane constituting the reference plane, and the transformation d(z)xe2x86x92Z being obtained by reverse application of the procedure for frontal perspective construction for reading metric distance from a standardised perspective drawing, and
determining on the basis of the established distance value d(z) and the element""s x,y co-ordinates in the two-dimensional image plane real spatial co-ordinates X,Y,Z for the element.
The invention comprises the creation of a reference scale by determining values of the parameters for colour by means of an image sequence of individual images, each of which depicts the subject in different predetermined focal planes, the changes in the colour parameters between the respective focal planes being used for calibration of a geometrical measuring scale which refers the respective measured colour parameters to metric or perspective distance values. The geometrical measuring scale preferably comprises geometrically coincident non-linear scales based on reference and empirical data for hue, colour saturation and brightness respectively. The colour temperature increases towards its maximum value at the common end point or point of convergence where the colour saturation and the contrast approach zero. The non-linear scales"" common point of convergence is determined by means of the colour parameters for one or more remotely located elements in the image which was focused on the focal plane for furthest distance, the measured values for this element or average measured values for these elements determining the point of convergence. In a further preferred embodiment the geometrical measuring scale is inserted in a frontal perspective diagram corresponding to the format of the two-dimensional image and with the non-linear scales calibrated for the colour of the image concerned in order to determine the distance between the reference plane and an element in the image.
Finally, according to the invention use is preferably made of either a perspective-based distance value or a metric distance value.
By this means exposed colour photographic image material is subjected to an analysis based on empirical and experimental knowledge of characteristics of colour, each colour thereby obtaining its perspective place relative to the image""s other colours. As a result the invention provides a three-dimensional reference database for all colours in the scenario, from which other perspective parameters such as distance, spatial co-ordinates etc. may be deduced.
In the present invention information is processed which exists in already exposed photographic colour pictures. If the photographic material is of the traditional analogue type, such as negative or positive colour film/paper or video, for further treatment it will be natural to convert the material to digital format, where each point in the image surface is represented by a pixel, i.e. a description of its characteristics, and which should contain amongst other things the point""s orientation in the image plane and the desirable data with regard to colour (hue, saturation and brightness). Such a digital representation can be provided by means of existing editing units for image processing, in which the invention may also be processed.
In the following description of the present invention a mathematical model is employed based on a geometrical method. The object of this embodiment is to provide the best possible explanation of the method. In practice it will be natural to implement the method in an alternative equivalent mathematical method for processing in a computer.